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COMPUTER-AIDED  AUTHORING  OF  INSTRUCTIONAL  MATERIALS 

/  Job  training  in  the  military  is  clearly  a  huge  enterprise.  For  example, 
the  Department  of  Defense  in  FY  78  provided  136,000  man-years  of  specialized 
skill  training  at  a  cost  of  $3.3  billion.  In  the  Navy  alone  there  are  over 
2,000  courses  on  the  calibration,  repair,  and  service  of  specific  pieces  of 
equipment.  Producing  job  training  materials  for  these  courses  is  expensive, 
and  the  training  effectiveness  of  these  materials  is  not  guaranteed  since 
effective  principles  derived  from  training  research  are  not  fully  incorporated 
into  these  materials.  Without  effective  training  materials,  costly  training 
time  is  wasted.  Currently,  the  problems  associated  with  writing  and  publishing 
instructional  materials  are  coming  under  systematic  scrutiny.  Several  large 
scale  design  efforts  for  the  efficient  presentation  of  technical  information 
are  underway. 

One  of  these  programs  is  the  Naval  Technical  Information  Presentation 
Program  (NTIPP),  a  5-year,  multimillion  dollar  program  to  improve  the  Navy's 
efficiency  in  publishing.  State-of-the-art  techniques  in  authoring,  editing, 
composing,  typesetting,  illustrating,  printing,  and  distributing  are  being 
applied  to  the  process  of  publishing  Navy  technical  information  to  support  the 
maintenance  and  operation  of  Navy  equipment.  This  effort  is  managed  by  the 
David  W.  Taylor  Naval  Ship  Research  and  Development  Center  with  participation 
by  other  Navy  activities.  The  Training  Analysis  and  Evaluation  Group  ~fT;AEG) 
represents  the  Chief  of  Naval  Education  and  Training  (CNET)  in  this  program. 

The  program  is  of  special  interest  to  CNET  because  the  technical  manuals  serve 
as  textbooks  in  the  CNET-managed  ?"CHV schools,  and  the  training  handbooks  serve 
as  the  student  and  instructor  guides  in  these  schools.  These  materials  also 
support  on-the-job  training.  Improved  documents  will  result  in  improved 
training. 

Within  NTIPP,  the  TAEG  is  investigating  the  feasibility  of  authoring 
instructional  materials  with  the  aid  of  computers.  Producing  effective 
materials  at  low  cost  is  the  major  goal.  Our  task^is  to  design  materials  to 
teach  the  operation  and  maintenance  procedures  for  military  equipment.  Ini¬ 
tially  we  have  developed  computer  routines  to  author  materials  for  teaching 
symbol  identification.  We  thought  this  would  be  a  simpler  computer  authoring 
task,  yet  would  provide  a  test-bed  for  some  of  our  ideas. 

A  single  symbol  learning  strategy  can  be  used  to  learn  all  types  of 
visually  perceived  symbols  such  as  weather  symbols,  electronic  symbols,  the 
Morse  code.  Navy  signal  flags,  acronyms,  and  abbreviations.  The  formats  and 
directions  used  in  presenting  information,  exercises,  and  tests  on  weather 
symbols  can  also  be  used  with  electronic  symbols  or  Morse  code  sight  patterns. 

The  symbol  learning  strategy  we  are  using  is  the  TAEG  algorithm  for 
identifying  symbols.  It  appears  in  the  Interservice  Procedures  for  Instruc¬ 
tional  Systems  Development  (1975)  and  in  TAEG  Report  No.  23,  Learning  Guide¬ 
lines  and  Algorithms  forTen  Types  of  Training  Objectives  (Aagard  and  Braby, 
1976) .  Formats  for  presenting  training  information  on  symbols  have  been  devel¬ 
oped  which  carry  out  the  intent  of  the  algorithm.  General  purpose  computers 
are  used  to  store  the  generic  formats  and  automate  the  writing  of  the  training 
materials.  With  this  system,  a  human  author  enters  information  about  a  set  of 
symbols  into  a  data  base.  This  data  base  is  far  simpler  to  write  (and  there- 
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fore  less  expensive)  than  the  writing  of  a  full  text.  The  computer  takes  over  at 
this  point,  which  is  possible  because  the  formats  of  this  kind  of  text  are  in 
many  ways  repetitive.  A  computer  routine  merges  the  contents  of  the  data  base 
with  the  generic  formats  and  automatically  organizes  and  prints  a  complete 
learning  package,  from  title  page  to  index  (see  figure  1). 

What  follows  is  a  description  of  the  computer-aided  authoring  technique 
for  generating  symbol  learning  materials.  In  addition,  three  related  issues 
are  discussed.  The  first  concerns  the  training  effectiveness  of  computer- 
authored  training  materials;  the  second  is  on  the  cost-effectiveness  of  authoring 
materials  in  this  manner;  the  third  deals  with  what  can  reasonably  be  expected 
from  computer  authoring  (i.e.,  can  we  expect  a  computer  to  author  materials  to 
support  major  areas  of  the  curriculum,  such  as  teaching  procedures,  classifying 
targets  or  tactical  decisionmaking). 

DESCRIPTION  OF  TECHNIQUE 

There  are  six  steps  in  our  current  form  of  computer-aided  authoring  (see 
figure  2). 

In  step  1,  the  human  author  supplies  the  information  called  for  in  an 
author's  workbook.  Information  required  includes  the  title,  authors,  publisher, 
publication  date,  training  objective,  a  statement  describing  the  importance  of 
knowing  the  specific  set  of  symbols,  and  the  examples  of  how  the  symbols  are 
used.  The  author  then  compiles  a  table  containing  each  symbol,  its  meaning, 
and  a  simple  memory  aid  (see  figure  3).  In  step  2,  this  information  is  keyed 
or  scanned  into  computer  memory.  At  this  time  the  computer  routines  create  the 
data  base  and  compose  all  the  pages  in  the  instructional  package  (steps  3  and 
4).  Included  are  introductory  pages,  symbol  discrimination  drills,  presentation 
of  symbols  and  meanings,  drills  for  recalling  symbols,  self-tests,  criterion 
tests,  and  suggestions  for  refresher  training.  This  is  fundamentally  different 
from  manual  word  processing  in  that  the  computer  automatically  pulls  data  out 
of  the  data  base  and  organizes  it  into  pages.  These  computer  routines  adapt  to 
a  wide  range  of  symbol  types  and  sizes. 

In  step  5  each  page  can  be  displayed  on  the  CRT  and  reviewed  for  accuracy, 
appropriateness  of  layout,  and  flow  of  information.  When  there  is  a  need  to 
edit  the  computer  composed  pages,  four  ways  are  available  to  change  the  material: 

edit  the  individual  pages  displayed  on  the  CRT 
edit  the  data  base  and  rerun  the  program 

edit  the  computer  program  (change  standard  headings,  margins,  etc.) 
and  rerun  the  program 

make  pencil  changes  to  a  computer  produced  manuscript  and  retype 
manually. 

Editing  is  an  optional  step,  in  that  you  could  produce  the  master  copy  for 
reproduction  at  this  time  without  further  review. 

When  the  electronically  stored  pages  are  in  an  acceptable  form,  the 
document  is  printed  (step  6)  onto  paper  by  the  use  of  a  line  printer,  computer 
driven  typewriter,  or  photo typesetter.  If  a  direct  microfiche  output  is 
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required,  or  if  graphic  symbols  are  included,  the  phototypesetter  should  be  used. 
Also,  the  phototypesetter  makes  it  possible  to  use  two  or  more  fonts  of  type 
on  a  page.  This  process  is  described  by  Braby,  Parrish,  Guitard,  and  Aagard 
(1978). 

The  automatic  authoring  software  was  developed  using  a  WANG  2200VP  mini¬ 
computer  system,  a  programmable  calculator.  This  hardware  was  used  because  it 
was  available.  The  central  processor  contains  32K  bytes  of  main  storage. 

This  small  amount  of  main  storage  made  it  necessary  to  use  program  overlays 
extensively  in  developing  the  software.  All  software  is  written  in  WANG  Basic 
which  is  an  extended  revision  of  Dartmouth  Basic  (Guitard,  1978).  While  the 
WANG  2200  is  probably  not  the  best  hardware  to  use  for  such  an  endeavor,  the 
Basic-2  language  available  with  the  WANG  2200VP  may  be.  WANG  Basic-2  has 
extensive  capabilities  for  manipulating  alphanumeric  strings.  The  equipment 
we  are  using  is  shown  in  figure  4.  The  software  can  be  adapted  for  use  on  the 
text  processing,  composing,  and  typesetting  equipment  planned  for  the  Naval 
Education  and  Training  Command  (Keeler,  1977). 

EFFECTIVENESS  IN  SUPPORTING  LEARNING.  Obviously,  anything  that  can  be  written 
by  a  computer  routine  can  also  be  written  by  a  human  author.  The  question  is: 

Can  we  devise  a  way  to  present  information  that  optimizes  learning  and  at  the 
same  time  is  machine  producible? 

We  put  this  question  to  the  test  by  comparing  computer  composed  materials 
with  traditional  materials  used  in  teaching  Morse  code  (Ainsworth,  1979).  The 
study  was  conducted  with  160  students  in  the  Signalman  "A"  School,  Orlando, 
Florida.  Four  types  of  instructional  material  were  compared. 

.  Traditional  Materials  consisted  of  a  set  of  flash  cards  and  a  six 
page  study  guide  containing  the  code  and  simple  exercises.  Students 
were  expected  to  develop  their  own  study  methods.  This  was  the 
approach  used  by  the  school  in  teaching  the  code. 

The  Mnemonics  Only  Handbook  contained  graphic  memory  aids  for  the 
letters  and  other  types  of  memory  aids  for  numbers  and  punctuation 
marks.  The  13  page  booklet  contained  no  practice  exercises  or  self¬ 
tests. 

The  Guided  Practice  Handbook  presented  the  Morse  code  in  six  sets. 
Practice  exercises  and  self-tests  were  included.  Mnemonics  were  not 
included.  This  137  page  handbook  was  produced  with  the  computer- 
aided  authoring  routines. 

The  Guided  Practice  with  Mnemonics  Handbook  was  similar  to  the 
Guided  Practice  Handbook  but  also  included  the  memory  aids  from  the 
Mnemonics  Only  Handbook.  Computer-aided  authoring  routines  were 
used  to  produce  this  material.  Sample  pages  from  SYMBOL  SET  # 2  are 
included  as  figure  5.  In  this  example,  the  computer  output  was 
edited,  and  the  graphic  symbols  were  added  by  hand. 
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The  intent  was  to  determine  the  differential  effects  of  aptitude,  type  of 
instructional  material  and  amount  of  study  time  on  the  a:quisition  of  Morse 
code.  A  2x4x3  repeated  measures  design  was  used,  involving  two  levels  of 
aptitude,  four  types  of  instructional  material,  and  three  periods  of  study. 

The  results  for  4  hours  of  study  are  shown  in  figure  6. 

In  Morse  code  receiving  is  more  difficult  to  learn  than  sending. 

Students  who  studied  receiving  Morse  code  with  the  two  types  of  handbooks 
produced  with  computer  aids  learned  significantly  better  ( p  <. 01 )  than  the 
students  who  studied  with  the  traditional  materials.  Similarly  students  who 
studied  sending  Morse  code  learned  significantly  better  (p<.01)  with  the 
Guided  Practice  With  Mnemonics  Handbook  (a  computer-generated  document)  and 
the  Mnemonics  Only  Handbook. 

The  overall  results  indicated  that  the  handbook  produced  with  the  full 
use  of  the  computer-aided  authoring  routines  (Guided  Practice  With  Mnemonics) 
was  superior  to  the  other  approaches.  This  handbook  essential ly  el iminated 
performance  differences  between  the  average  and  above  average  groups.  After 
4  hours  of  study  using  the  computer  authored  text  with  mnemonics  and  guided 
practice,  the  combined  score  (for  sending  and  receiving)  was  93%  for  the 
average  group  and  96%  for  the  above  average  group.  When  traditional  materials 
were  used  there  was  a  large  difference  in  performance  between  the  two  groups. 
The  combined  score  for  the  average  group  was  75%  and  89%  for  the  above  average 
group.  (Students  in  the  average  group  had  scores  between  99  and  114  on  the 
Word  Knowledge  and  Arithmetic  Reasoning  subtests  of  the  Armed  Services  Voca¬ 
tional  Aptitude  Battery  (ASVAB).  Students  in  the  above  average  group  had 
scored  between  115  and  143  on  these  tests.)  As  the  result  of  this  evaluation, 
the  Guided  Practice  With  Mnemonics  Handbook  has  been  adopted  by  the  Navy 
Signalman  Schools,  and  all  new  signalmen  will  use  it  in  their  study  of  Morse 
code. 


Clearly  our  computer  authoring  routines  worked  well  in  this  particular 
case.  The  results  indicated  the  feasibility  of  using  computer-aided  authoring 
routines  for  producing  effective  instructional  materials  for  symbol  recognition. 

COST  EFFECTIVENESS.  Computer  authoring  can  work  well  but  at  what  cost?  The 
use  of  computer-aided  authoring  appears  to  be  a  low  cost  way  of  producing 
these  materials,  but  the  facts  are  not  all  in.  We  are  currently  preparing  a 
report  that  discusses  the  costs  and  compares  them  with  the  cost  of  the  tra¬ 
ditional  labor  intensive  way  of  authoring  these  materials.  Our  observations 
to  date  include  the  following. 

1.  Creating  the  data  base  is  a  relatively  simple  and  inexpensive  task. 

2.  Once  the  data  base  exists,  it  takes  less  than  5  minutes  of  machine 
time  to  produce  a  100  page  book. 

3.  Proofreading,  changing  text,  and  retyping  with  the  computer-aided 
system  saves  hours  of  production  time.  For  instance,  once  the  information  in 
the  data  base  is  spelled  correctly,  the  computer  generated  handbook  will  con¬ 
tain  no  typographical  errors  or  misspelled  words. 
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BROAD  USE  IN  CURRICULUM  DEVELOPMENT.  Can  computer-aided  authoring  routines 
support  major  curriculum  areas?  We  are  attempting  to  find  the  answer  to  this 
question.  It  seems  reasonable  to  expect  that  computer  routines  will  be  of 
real  value  in  authoring  materials  with  high  levels  of  redundancy,  such  as 
found  in  technical  training  materials.  These  training  materials  often  present 
the  same  information  in  overviews,  descriptions,  self-tests,  summaries, 
criterion  tests  with  answers,  and  refresher  training  guides.  In  addition, 
standard  writing  and  presentation  formats  are  used  repetitively  in  these 
materials.  These  content  and  format  redundancies  are  not  found  in  most  other 
types  of  writing. 

TAEG  is  now  designing  computer-aided  authoring  routines  for  material  to 
teach  equipment  operation  and  maintenance  procedures  to  be  performed  from 
memory.  We  are  well  into  this  task.  It  involves  extensive  use  of  digitized 
line  drawing  and  photographs  as  well  as  alphanumerics  and  is  a  logical  exten¬ 
sion  of  our  work  with  symbols.  If  successful,  these  routines  could  support 
major  segments  of  the  Navy's  "A"  and  "C"  schools  curricula  and  may  support  the 
production  of  some  types  of  job  performance  aids. 

Other  types  of  job  tasks  being  considered  are  classifying/recognizing 
patterns,  rule  using,  tactical  decisionmaking,  recalling  bodies  of  knowledge, 
and  voice  communicating. 

READABILITY  EDITING.  Another  aid  to  the  author  being  developed  by  TAEG  is  a 
readability  editing  system  (Kincaid,  1978).  While  this  is  a  separate  system, 
it  will  be  used  in  editing  the  data  base  in  the  computer-aided  authoring 
system.  This  system  is  being  developed  for  use  in  writing  centers  with 
computer  processing  capabilities.  As  draft  training  material  is  computer 
processed,  much  useful  feedback  is  provided  the  technical  writer  and  his 
editor  for  revising  drafts.  There  are  three  components  to  the  system: 

1.  A  feature  to  flag  "uncommon"  wo^ds.  These  are  defined  as  those  not 
on  a  list  of  1,900  common  words  which  was  compiled  by  a  frequency  analysis  of 
240,000  words  of  recruit  training  material. 

2.  The  calculation  of  grade  level  of  readability  according  to  the 
Department  of  Defer se  readability  formula  (Kincaid,  Fishburne,  Rogers,  and 
Chissom,  1975). 

3.  A  word  and  phrase  substitution  :eature.  Awkward  phrases  such  as  "in 
accordance  with"  are  flagged  and  substitutes  such  as  "according  to"  are 
suggested. 

A  computer  printout  incorporating  each  of  these  features  is  almost 
immediately  available  to  the  technical  writer.  Perhaps  the  most  useful 
feature  of  the  system  is  that  glossaries  can  be  compiled  based  on  the  uncommon 
words  that  are  flagged. 
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POSTNOTE 

The  results  of  the  computer-aided  authoring  study  to  date  have  been 
impressive.  The  use  of  computer  aids  appears  to  be  an  effective  and  low  cost 
way  of  producing  training  materials.  But  this  is  just  the  beginning.  Evidence 
is  mounting  that  basic  instructional  design  processes  can  be  automated,  and  it 
now  appears  feasible  to  develop  these  computer-based  authoring  aids  to  support 
the  performances  required  in  a  broad  range  of  job  tasks. 
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Figure  1.  The  Sequence  of  Page  Formats  Used  in  the  Programmed  Texts 
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An  Automatic  Authoring  System  -  Data  Entry  Workbook  Page  1 

This  workbook  outlines  the  information  that  you,  the  author,  must  supply  to 
create  an  instructional  module  for  teaching  symbol  recognition.  In  providing 
this  information,  follow  the  directions  carefully.  Record  your  Information 
in  the  spaces  provided. 

1.  Enter  the  following  information  for  the  title  page: 

1.1  «*  Title:  SASIC  SXEECU&S  IM  MQgS£COD£ 


1.2 

1.3 


Max: 

Subtitle: _ 

Max: 

**  Author(s): 


2  lines,  45  characters  per  line. 


2  lines,  50  characters  per  line. 


'RlfiUftgD  B1?ABV _ 

.imes  AIMSIA/OPTU 


Max:  4  lines,  50  characters  per  line. 

1.4  Date  wanted  on  title  page:  JANUAttV  1373 _ 

Max:  1  line,  50  characters  per  line. 

1.5  **  Organization  producing  the  program:  _ 

SEEV'CE  SP.UOOU  COMMAND _ 

QgLhNPO.fLnttim  32RI3 _ 

Max:  2  lines,  50  characters  per  line. 

2.  **  Record  your  learning  objective(s) :  mss.  C0MPUSTIH(t  this 

-pgQ&SAM  you  WILL.  BE  ABLE  TO  WRITE  MORSE- _ 

OffP£  EQUIVALENTS  OP  LETTERS,  NUMBERS,  AMP _ 

•PlJMCTUATIflN  MASKS _ 

**  See  section  2.0  of  this  report  for  special  formatting  techniques. 

Figure  3.  Sample  Pages  from  the  AUTHOR  Workbook 
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AUTHOR:  An  Automatic  Authoring  System  Data  Entry  Workbook:  Symbols 
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SYMBOL  SET  J 

2:  LEARN  SYMBOL  DEFINITIONS 

Definition  I  Memory  Aid 


ALFA  -  alphabet  soup 

UNIFORM  Insignia 

VICTOR  in  boxing 

NOVEMBER  turkey 

DELTA  jet  and  2  clouds  < 

BRAVO!  Play  3  notes 
again! 

JULIETT 


T 


j‘.V 


Go  to  19 


Figure  5.  Sample  Pages  from  the  Guided  Practice 
With  Mnemonics  Handbook  for  Morse  Code 
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PRACTICE  DEFINING  THE  SYMBOLS _  21 

1.  Read  all  directions  before  you  practice. 

2.  On  page  <22,  recall  the  Memory  Aid  for 
each  Morse  Code  symbol  in  the  "Practice 
Symbols"  section.  Then  recall  the  letter 
associated  with  that' code.  WRITE  the  letter 
on  scratch  paper. 

3.  CHECK  your  answers  after  writing  them 
down.  The  answers  are  below  the  practice 
symbols. 


Go  to  22 


Figure  5.  Sample  Pages  from  the  Guided  Practice  With  Mnemonics 
Handbook  for  Morse  Code  (continued) 


SYMBOL  SET  #2: 
Directions 
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